Study 4 Introduction

This Markdown documents the process to analyze the data for Study 4 which looked at the difference between those who received and those who did not receive a warning at the beginning of the study and the effects of this warning after a 1 day delay.

Data was collected between August 7, 2018 and August 9, 2018 by using Amazon’s Mechanical Turk for ditribution and Qualtrics as a survey platform.

This HTML was last knitted on: 2019-09-16 22:02:12

Set Up

Packages and Libraries

You must run this section before you can run any other chunks.

Data Import

Exclusions

You can exclude subjects who did not get the correct answer in the exercise by changing Exclude_Exercise_Check (line 99) to TRUE. The next time you run all the code, these participants will be excluded.

For this report, no participants are being excluded from analysis.

Methods

Participants

157 (53 women, Mage = 33.92, SDage = 9.97) completed both sessions of Study 4.

175 people completed session 1. 157 returned to complete session 2. 89.7142857142857% return rate.

All participants reported the United States as their location and had a previous task approval rate that was equal to or exceeded 85%.

74% reported having at least a Bachelor’s degree. The samples also exhibited a range of graph literacy (see Table 1).

Achieved power for main effects

## 
##      Paired t test power calculation 
## 
##               n = 77
##               d = 0.4
##       sig.level = 0.05
##           power = 0.9338806
##     alternative = two.sided
## 
## NOTE: n is number of *pairs*

Exercise Check

79 people did not complete exercise since they were in the NO WARNING condition. 2 other people did not complete the exercise at all.

16 did not get the manipulation check question right. Accuracy was 76.923%

For this report, no participants are being excluded from analysis. If you want to see results when participants who got the exercise wrong are excluded/included, you can go to the section called Exclusions (at the top of this file) and change Exclude_Exercise_Check <- TRUE

Results

The truncation effect

We first replicated our central effect of interest: the truncation effect.

We found that average ratings for truncated graphs was higher than ratings for control graphs: Mcontrol = 3.78, SD = 0.99; Mtruncated = 4.59, SD = 0.9.

## 
## Cohen's d
## 
## d estimate: -0.5338497 (medium)
## 95 percent confidence interval:
##      lower      upper 
## -0.7598465 -0.3078529
## 
##  Paired t-test
## 
## data:  subject_mean_rating by graph_condition
## t = -15.972, df = 156, p-value < 0.00000000000000022
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -0.9168274 -0.7150198
## sample estimates:
## mean of the differences 
##              -0.8159236

This main effect was statistically significant: t(156) = 15.97, p < 1.204459e-34, 95% CI of difference = [0.72, 0.92], d = 0.53.

Most participants in both sessions showed an overall truncation effect: 88.54% of participants in Session 1 (139 of 157) and 85.35% of participants in Session 2 (139 of 157) .

The truncation effect across warning conditions

Does an explanatory warning reduce the size of the truncation effect, as seen in previous studies? To interrogate an observed interaction between graph condition and warning condition , we computed models for each warning condition separately. In these two models, graph type (0 = control, 1 = truncated) served as a binary fixed factor, and the judged difference between bars was the outcome variable. Participant and item were included as random effects.

No warning

In the no warning condition, we found a significant difference between control and truncated graphs (b = 1.08, SE = 0.03, t = 33.48, p < 4.824725e-226) such that truncated graphs, relative to control graphs, exaggerated the judged difference between bars. In the warning condition,

Warning condition

In the warning condition, we also found a significant difference between control and truncated graphs (b = 0.55, SE = 0.03, t = 16.49, p < 7.812787e-60) such that truncated graphs, relative to control graphs, exaggerated the judged difference between bars. Nevertheless, the magnitude of the judged difference between bars is larger in the no warning condition than in the warning condition.

Figure 6

Is an explanatory warning still protective 24 hours later?

To answer this, we computed linear models for Session 1 and Session. In these two models, graph type (0 = control, 1 = truncated) and warning condition (0 = no warning, 1 = warning) served as binary fixed factors, and the judged difference between bars was the outcome variable. Participant and item were modeled as random effects.

In session 1, we found a statistically significant main effect of graph type (b = 1.15, SE = 0.05, t = 24.85, p < 6.681194e-130) and a statistically significant interaction between graph type and warning condition (b = -0.59, SE = 0.07, t = -9, p < 3.061309e-19).

Critically, we saw a similar pattern of results in session 2: we found a statistically significant main effect of graph type (b = 1.01, SE = 0.05, t = 22.3, p < 6.061202e-106) and a statistically significant interaction between graph type and warning condition (b = -0.47, SE = 0.06, t = -7.29, p < 3.489533e-13).

Overall, these results (summarized in Figure 6) show that an explanatory warning results in a smaller truncation effect, and that this protective effect is still present 24 hours later.

Supplemental Information

Methods (Supplemental Information)

Participants (Supplemental Information)

156 participants reported English as their first language, and 1 reported Spanish as their first language.

74% of participants reported having at least a Bachelor’s degree.

Session 1 of the experiment took participants an average of Mduration = 18.31 minutes, (SDduration = 23.68 minutes).

Session 2 of the experiment took participants an average of Mduration = 18.82 minutes, (SDduration = 21.24 minutes).

Results (Supplemental Information)

Full model (Supplemental Information)

Education (Supplemental Information)

Education and graph literacy

## 
## Call:
## lm(formula = graphliteracy_sum_rating ~ dem_ed, data = models_difference_df)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -23.7778  -3.4636   0.5364   5.2222  16.3007 
## 
## Coefficients:
##             Estimate Std. Error t value             Pr(>|t|)    
## (Intercept)  40.9349     1.2652  32.354 < 0.0000000000000002 ***
## dem_ed        0.9215     0.2328   3.958            0.0000938 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 6.878 on 312 degrees of freedom
## Multiple R-squared:  0.0478, Adjusted R-squared:  0.04475 
## F-statistic: 15.66 on 1 and 312 DF,  p-value: 0.00009379

Graph Literacy (Supplemental Information)

Main Effect

Graph literacy does not explain the size of the truncation effect

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: difference ~ graphliteracy_sum_rating + (1 | participantid)
##    Data: models_difference_df
## 
## REML criterion at convergence: 596.7
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -2.6466 -0.5339 -0.0452  0.6059  1.9665 
## 
## Random effects:
##  Groups        Name        Variance Std.Dev.
##  participantid (Intercept) 0.3224   0.5678  
##  Residual                  0.1753   0.4187  
## Number of obs: 314, groups:  participantid, 157
## 
## Fixed effects:
##                            Estimate Std. Error         df t value Pr(>|t|)
## (Intercept)                1.126365   0.336297 154.999928   3.349  0.00102
## graphliteracy_sum_rating  -0.006793   0.007273 154.999928  -0.934  0.35178
##                            
## (Intercept)              **
## graphliteracy_sum_rating   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## grphltrcy__ -0.988

Truncation effect is mean rating for truncated graphs - mean rating for control graphs.

Interaction with warning

Graph literacy does not significantly predict the size of the truncation effect even when you add in warning condition.

## 
## Call:
## lm(formula = difference ~ graphliteracy_sum_rating * subject_condition, 
##     data = models_difference_df)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -1.66441 -0.46085  0.01493  0.44796  1.76937 
## 
## Coefficients:
##                                                    Estimate Std. Error
## (Intercept)                                        1.348402   0.348808
## graphliteracy_sum_rating                          -0.005852   0.007443
## subject_conditionwarning                          -0.132132   0.486116
## graphliteracy_sum_rating:subject_conditionwarning -0.008909   0.010514
##                                                   t value Pr(>|t|)    
## (Intercept)                                         3.866 0.000135 ***
## graphliteracy_sum_rating                           -0.786 0.432350    
## subject_conditionwarning                           -0.272 0.785947    
## graphliteracy_sum_rating:subject_conditionwarning  -0.847 0.397467    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.6518 on 310 degrees of freedom
## Multiple R-squared:  0.1523, Adjusted R-squared:  0.1441 
## F-statistic: 18.56 on 3 and 310 DF,  p-value: 0.00000000004241

Truncation effect is mean rating for truncated graphs - mean rating for control graphs.

Interaction with session

Graph literacy does not significantly predict the size of the truncation effect even when you add in warning session.

Truncation effect is mean rating for truncated graphs - mean rating for control graphs.

Age (Supplemental Information)

Main Effect
## 
## Call:
## lm(formula = difference ~ dem_age, data = models_difference_df)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2.00428 -0.54541 -0.04714  0.48901  1.73727 
## 
## Coefficients:
##             Estimate Std. Error t value   Pr(>|t|)    
## (Intercept) 0.698459   0.141439   4.938 0.00000129 ***
## dem_age     0.003463   0.004001   0.865      0.387    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.7048 on 312 degrees of freedom
## Multiple R-squared:  0.002395,   Adjusted R-squared:  -0.0008027 
## F-statistic: 0.7489 on 1 and 312 DF,  p-value: 0.3875

Truncation effect is mean rating for truncated graphs - mean rating for control graphs.

Interaction with warning
## 
## Call:
## lm(formula = difference ~ dem_age * subject_condition, data = models_difference_df)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -1.72211 -0.48106  0.00371  0.45433  1.55739 
## 
## Coefficients:
##                                   Estimate Std. Error t value  Pr(>|t|)
## (Intercept)                       0.792698   0.197199   4.020 0.0000732
## dem_age                           0.008508   0.005688   1.496     0.136
## subject_conditionwarning         -0.309088   0.264946  -1.167     0.244
## dem_age:subject_conditionwarning -0.006541   0.007515  -0.870     0.385
##                                     
## (Intercept)                      ***
## dem_age                             
## subject_conditionwarning            
## dem_age:subject_conditionwarning    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.654 on 310 degrees of freedom
## Multiple R-squared:  0.1464, Adjusted R-squared:  0.1381 
## F-statistic: 17.72 on 3 and 310 DF,  p-value: 0.0000000001218

Truncation effect is mean rating for truncated graphs - mean rating for control graphs.

Interaction with session
## 
## Call:
## lm(formula = difference ~ dem_age * session, data = models_difference_df)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -1.96750 -0.54005 -0.04463  0.51283  1.77872 
## 
## Coefficients:
##                    Estimate Std. Error t value Pr(>|t|)    
## (Intercept)       0.7527322  0.2003225   3.758 0.000205 ***
## dem_age           0.0030738  0.0056668   0.542 0.587920    
## session2         -0.1085468  0.2832988  -0.383 0.701869    
## dem_age:session2  0.0007777  0.0080141   0.097 0.922760    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.7058 on 310 degrees of freedom
## Multiple R-squared:  0.005837,   Adjusted R-squared:  -0.003784 
## F-statistic: 0.6067 on 3 and 310 DF,  p-value: 0.6111

Truncation effect is mean rating for truncated graphs - mean rating for control graphs.

Item Analysis (Supplemental Information)

Main Effect of Warning

Here we summarise the data for each item. This is helpful to identify if the effect is driven by a few graphs or if it is seen across all graphs in the materials.

The effect of warning was observed for all graphs except graph 2, 12 and 40. On average, the 7-point ratings were 0.15 (SD = 0.19) higher when graphs were seen in the no warning condition than when they were seen in the warning condition. The maximum average absolute difference between the no warning and the warning conditions was for graph 7 M = 0.54. The minimum average absolute difference was for graph 2 where M = 0.02.

Warning interaction with graph type

Session interaction with graph type

Q: Does anything change if we put all the things in?

## 
## Call:
## lm(formula = difference ~ dem_age + dem_ed + graphliteracy_sum_rating + 
##     session + subject_condition, data = df)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -1.63392 -0.41407  0.01969  0.40114  1.66396 
## 
## Coefficients:
##                           Estimate Std. Error t value        Pr(>|t|)    
## (Intercept)               1.712390   0.296611   5.773 0.0000000189845 ***
## dem_age                   0.005238   0.003619   1.447        0.148802    
## dem_ed                   -0.086343   0.022317  -3.869        0.000133 ***
## graphliteracy_sum_rating -0.005558   0.005277  -1.053        0.293014    
## session                  -0.082166   0.071817  -1.144        0.253475    
## subject_conditionwarning -0.503885   0.072942  -6.908 0.0000000000282 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.6363 on 308 degrees of freedom
## Multiple R-squared:  0.1972, Adjusted R-squared:  0.1842 
## F-statistic: 15.13 on 5 and 308 DF,  p-value: 0.0000000000002692

Timing (Supplemental Information)

Timing information is in seconds.

You can choose to trim or not timing data. By default any timing that is 2 standard deviations away from the mean (for each participant for each condition) is trimmed.

For this report, any timing that is 2 standard deviations away from the mean (for each participant for each condition) WAS NOT trimmed. If you want to see results when participants who got the exercise wrong are excluded/included, you can go to the section called Exclusions (at the top of this file) and change Trimming <- FALSE